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ABSTRACT 


Numerous  reform  initiatives  have  attempted  to  improve  the 
acquisition  process.  The  success  or  failure  of  these 
initiatives  have  often  been  based  on  subjective 
determinations.  In  order  to  determine  the  true  effect  of 
these  initiatives  we  must  be  able  to  measure  the  effect  of 
these  initiatives  on  the  acquisition  process.  Measurement 
requires  the  development  of  metrics.  This  study  explores  the 
use  of  metrics  for  acquisition  reform,  using  the  Defense 
Acquisition  Workforce  Improvement  Act  (DAWIA) ,  Public  Law  101- 
510,  as  a  case  study. 

This  study  identifies  the  objectives  of  DAWIA.  Using  the 
Policy  Effectiveness  Model,  the  study  develops  and  proposes 
metrics  for  DAWIA  objectives  in  an  effort  to  measure  the 
implementation  and  effectiveness  of  this  important  and  far- 
reaching  acquisition  reform  legislation. 
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I .  INTRODUCTION 


A.  AREA  OF  RESEARCH 

The  purpose  of  this  research  is  to  explore  how  the 
effects  of  the  Defense  Acquisition  Workforce  Improvement  Act 
(DAWIA)  of  1990  can  be  measured. 

B .  BACKGROUND 

The  Defense  Acquisition  Workforce  Improvement  Act  is  the 
public  law  enacted  in  1990  that  prescribes  how  the  acquisition 
workforce  will  be  managed.  The  objective  of  DAWIA  is  to 
develop  a  dedicated  pool  of  highly  qualified  military  and 
civilian  acquisition  specialists  to  fill  designated  critical 
acquisition  positions  in  order  to  create  a  more  efficient  and 
effective  procurement  system  for  the  Government.  DAWIA  covers 
acquisition  corps  membership  requirements,  contracting  officer 
requirements,  PM  qualification  standards,  requirements  for 
assignment  to  critical  acquisition  positions,  and  waiver 
requirements . 

To  date,  a  great  deal  of  time,  effort,  and  money  have 
been  spent  on  implementing  the  requirements  of  DAWIA. 
However,  there  is  little  information  available  indicating  how 
the  acquisition  process  has  benefitted  as  a  result  of  that 
time,  effort,  and  money.  This  research  explores  ways  to 
determine  those  benefits.  Specifically  it  looks  at  ways  to 
determine  DAWIA 's  impact  on  the  acquisition  process  and 
introduces  metrics  as  measures  of  effectiveness. 

Measurement  often  leads  to  controversy  and  argument . 
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What  are  appropriate  metrics  for  the  process  and  the  product? 
How  should  the  data  be  collected  and  used?  Is  it  fair  to  use 
measurements  to  compare  people,  processes,  or  products?  These 
questions  and  dozens  of  others  always  surface  when  an  attempt 
is  made  to  measure  something  that  has  not  been  measured  in  the 
past.  Such  is  the  case  with  measuring  the  effect  of  DAWIA. 

C .  RESEARCH  QUESTIONS 

The  primary  research  question  is: 

How  can  the  effectiveness  of  DAWIA  be  measured? 
Subsidiary  research  questions  are: 

1.  What  are  the  objectives  of  DAWIA? 

2 .  What  is  a  metric  and  what  types  of  metrics  are  suitable 
for  acquisition? 

3 .  What  metrics  are  currently  being  used  to  measure  the 
effects  of  DAWIA  on  the  acquisition  process? 

4.  Are  these  appropriate  metrics? 

5.  Are  there  standardized  metrics  for  acquisition  reform 
initiatives?  If  not,  should  standards  be  established? 

D.  SCOPE 

This  thesis  describes  DAWIA,  a  history  of  acquisition 
reform  and  the  events  that  led  to  DAWIA;  discusses  the  use  of 
metrics  to  measure  people,  processes,  and  products;  determines 
and  discusses  the  metrics  used  to  measure  the  effect  of  DAWIA; 
and  provides  recommendations. 
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E .  METHODOLOGY 


The  first  objective  of  this  research  is  to  provide  an 
overview  of  acquisition  reform,  DAWIA,  and  metrics.  This  is 
accomplished  through  a  literature  review  of  sources  including: 

-  References,  publications  and  electronic  media  available 
at  the  Naval  Postgraduate  School  Library 

-  Internet  websites  and  homepages  (DoD,  commercial,  and 
academic) 

-  Published  academic  research  papers 

-  Unclassified  Department  of  Defense  publications 

The  second  objective  is  to  analyze  DAWIA  and  determine 
suitable  metrics  for  measuring  the  effectiveness  of  DAWIA. 
The  analysis  will  be  accomplished  using  the  Policy 
Effectiveness  Model  [Ref.  1:  p.  330].  The  primary  sources  of 
this  information  are  DoD  reports,  acquisition  literature,  and 
interviews  with  faculty  and  officials  from  Defense  Acquisition 
University,  Defense  Systems  Management  College,  and  the  Naval 
Postgraduate  School;  acquisition  reform  office  personnel;  and 
acquisition  workforce  personnel . 

F .  ORGANIZATION 

Chapter  II  (Background)  provides  a  history  of  acquisition 
reform  and  describes  the  events  that  led  to  enactment  of 
DAWIA.  It  also  provides  a  description  of  DAWIA  requirements 
and  objectives. 

Chapter  III  (Metrics)  introduces  metrics,  providing 
definition,  purpose,  and  benefits.  The  chapter  describes  a 
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methodology  for  establishing  metrics  in  an  organization.  The 
chapter  then  examines  the  use  of  metrics  in  various  fields- - 
software,  education,  and  DoD  acquisition  reform. 

Chapter  IV  (Analysis  of  DAWIA  Metrics)  provides  an 
analysis  of  the  metrics  used  for  DAWIA  and  an  analysis  of 
DAWIA  objectives.  The  chapter  then  tailors  metrics  to  DAWIA 
requirements . 

Chapter  V  (Summary,  Conclusions,  and  Recommendations) 
summarizes  the  findings  of  the  research,  answers  the  research 
questions,  and  presents  recommendations  for  further  research 
and  study. 

G.  BENEFITS  OF  STUDY 

This  study  explores  ways  to  determine  the  effectiveness 
of  DAWIA  through  the  use  of  metrics.  This  information 
provides  valuable  feedback  to  acquisition  leaders  that  will 
enable  them  to  develop  measures  and  techniques  for  determining 
the  effectiveness  of  acquisition  reform  initiatives.  The  use 
of  measures  allows  acquisition  leaders  to  assess  DAWIA  and 
possibly  other  reform  initiatives  in  order  to  direct  future 
efforts  and  give  them  opportunity  to  provide  maximum  benefit 
to  the  acquisition  process. 
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II.  BACKGROUND 


A.  INTRODUCTION 

This  chapter  provides  a  history  of  acquisition  reform  and 
describes  the  events  that  led  to  the  enactment  of  DAWIA.  It 
also  provides  a  description  of  DAWIA  requirements  and  closes 
with  a  chapter  summary. 

B.  HISTORY  OF  ACQUISITION  REFORM 

The  federal  acquisition  system  is  undoubtedly  one  of  the 
most  thoroughly  developed,  documented,  examined,  and 
scrutinized,  acquisition  systems  in  existence  [Ref.  2:  p.  2]. 
The  federal  acquisition  system  has  long  suffered  from  negative 
perceptions.  Innumerable  articles  and  reports  have  been 
generated  discussing  the  ineffectiveness  of  our  acquisition 
system.  Reports  of  the  huge  cost  overruns  involving  the 
purchases  of  weapons,  vehicles,  hammers,  etc.  have  filled  the 
newspapers  for  decades . 

As  a  result,  there  have  been  numerous  attempts  to  improve 
the  acquisition  process  through  various  reform  initiatives. 
These  reform  initiatives  began  with  the  large  procurement  of 
weapons  in  World  War  II  and  are  likely  to  continue  as  long  as 
the  government  remains  in  the  procurement  business. 

Weapons  production  in  the  United  States  has  been  big 
business  since  World  War  II.  President  Roosevelt,  recognizing 
the  need  to  mobilize  the  nation's  industrial  might,  sought  to 
centralize  weapons  procurement  when  he  appointed  Donald  Nelson 
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as  the  War  Production  board  Chairman.  Nelson's  mission  was 


to  oversee  the  military  department's  development  and  purchase 
of  war  equipment.  Nelson,  however,  failed  to  carry  through 
with  Roosevelt's  intent  of  centralizing  procurement.  He 
elected  to  defer  to  the  departments,  believing  his  decisions 
would  undermine  their  process  with  unnecessary  civilian 
oversight.  Nelson’s  hands-off  approach  likely  resulted  from 
his  predecessor,  Bernard  Baruch,  who  upon  being  appointed 
during  World  War  I  by  President  Wilson,  was  directed  to 
exercise  minimal  interference . [Ref .  1] 

The  current  acquisition  management  system  began  to 
develop  in  the  Eisenhower  administration.  After  World  War  II, 
there  was  rapid  development  of  technology  and  a  tremendous 
increase  of  worldwide  American  national  security  commitments. 
As  a  result,  traditional  roles  of  the  services  were  split, 
creating  interservice  rivalry  over  weapon  system  development 
responsibility.  President  Eisenhower's  answer  to  interservice 
rivalry  and  the  resulting  competition  for  resources  was  the 
creation  of  a  centralized  civilian  authority  in  the  relatively 
new  Department  of  Defense . [Ref .  3:  p.  2] 

Eisenhower's  proposal  sought  to  establish  a  single 
uniformed  service  and  to  restructure  the  component  branches 
along  functional  lines.  However,  because  it  ran  contrary  to 
the  prerogatives  of  the  uniformed  services  and  perhaps 
Congress,  this  proposal  was  not  adopted.  [Ref.  3:  p.  2] 

Afterwards,  Eisenhower's  1958  Defense  Reorganization  Act 
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realigned  the  U.S.  defense  structure,  but  unfortunately- 
separated  the  administrative  functions  from  the  responsibility 
structure.  This  separation  further  complicated  an  already 
complex  bureaucracy. [Ref.  3:  p.  3] 

During  this  era,  contracting  was  dominated  by  cost-plus 
contracts  in  an  effort  to  push  technology  forward  rapidly  and 
gain  on  the  Soviets.  However,  in  the  1960 ' s  the  use  of  cost- 
plus  contracts  fell  into  disrepute  as  huge  overruns  occurred. 

In  1961,  Secretary  of  Defense  (SECDEF)  Robert  McNamara 
sought  to  reform  the  acquisition  process  by  instituting  a 
business  school  approach  to  analyzing  the  needs  of  the 
nation's  defense.  McNamara  wanted  to  incorporate  more  control 
into  the  acquisition  process  and  he  wanted  the  process  to 
provide  all  of  the  information  leaders  needed  in  order  to  make 
decisions.  He  brought  in  Charles  Hitch  from  the  RAND 
Corporation  to  develop  a  systematic  process  for  establishing 
requirements  and  incorporating  them  into  a  five-year  budget. 
Hitch  developed  this  process  which  later  became  known  as  the 
Planning,  Programming,  and  Budgeting  System  (PPBS).[Ref.  3:  p. 
3] 

McNamara's  attempts  to  improve  poor  management  through 
innovations  in  program  planning,  source  selection, 
contracting,  and  program  management  failed  to  achieve  expected 
results.  His  Total  Package  Procurement  (TPP) ,  essentially  a 
fixed-price  contract  for  Research  and  Development  (R&D)  and 
initial  procurement,  failed  to  achieve  expected  results.  TPP 
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programs  such  as  the  C-5A,  F-111A,  and  the  F-14  failed  to 
constrain  cost,  eventually  had  to  be  rewritten,  and  brought 
unfavorable  nationwide  attention . [Ref .  3:  p.  4]  As  a  result, 
fixed-price  contracts  fell  into  disuse  on  major  systems  until 
the  1980s. 

During  the  Nixon  and  Ford  administrations.  Secretary  of 
Defense  Melvin  Laird  returned  some  of  the  services'  autonomy, 
but  retained  some  central  control  through  a  new  senior  level 
board  called  the  Defense  Systems  Advisory  Review  Council 
(DSARC) .  The  DSARC  and  the  Cost  Analysis  Improvement  Group 
(CAIG)  provided  the  SECDEF  more  oversight  over  weapons 
acquisitions.  Other  attempts  at  improvements  were  short-lived 
or  they  were  implemented  superficially  and  for  the  most  part 
were  considered  unsuccessful . [Ref .  3:  p.  4] 

The  cycle  of  change  continued  with  the  next 
administration  when  Harold  Brown,  SECDEF  in  the  Carter 
administration,  sought  to  regain  some  of  the  authority  in 
weapons  acquisition  relinquished  previously.  He  issued  a 
requirement  for  the  services  to  comply  with  Circular  A-109,  an 
directive  published  by  the  Office  of  Federal  Procurement 
Policy  in  1976.  Circular  A-109  required  the  services  to 
prepare  a  mission  area  analysis  and  document  their  weapons 
need  in  a  mission  needs  statement. [Ref.  3:  p.  4] 

Change  in  the  acquisition  process  continued  with  the  next 
administration.  In  1981,  Caspar  Weinberger,  the  SECDEF  in  the 
Reagan  administration,  implemented  a  change  that  reversed  the 
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trend  towards  centralization.  Weinberger  gave  more  authority 
to  subordinate  line  executives,  especially  program  managers, 
to  execute  OSD  policy. [Ref.  3:  p.  4] 

Twenty  years  after  McNamara,  the  acquisition  process  was 
more  structured  and  complex.  In  1981,  Frank  Carlucci, 
Weinberger's  Deputy  Secretary  of  Defense,  directed  the 
services  to  implement  32  initiatives  to  reform  the  acquisition 
process.  These  initiatives  aimed  to  streamline  the 
acquisition  process,  reprogram  costs,  and  shorten  acquisition 
time.  The  underlying  principle  of  the  Carlucci  initiatives 
was  that  over- regulation  undermined  efficiency.  The  services 
started  many  of  his  initiatives,  but  success  was  short-lived 
due  to  three  factors;  1)  a  Congress  reluctant  to  relinquish 
some  of  its  purse -string  powers,  2)  services  reluctant  to 
change  some  of  their  practices,  and  3)  Carlucci's  departure 
due  to  a  new  administration. [Ref.  3:  p.  4] 

Until  the  mid-eighties  most  reform  recommendations 
addressed  DoD's  acquisition  organization  or  process.  In  1986, 
Reagan's  Presidential  Blue  Ribbon  Panel  on  Defense  Management, 
also  known  as  the  Packard  Commission,  continued  with  this 
trend.  Procurement  spending  doubled  between  1980  and  1985, 
resulting  in  increased  attention  on  defense  acquisition  and 
further  examination  of  the  process.  The  Commission,  after 
examining  the  process,  concluded  that  the  defense  acquisition 
process  was  not  being  operated  and  managed  effectively,  and 
was  negatively  affecting  cost  and  efficiency  of 
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procurement.  [Ref.  3:  p.  5] 

As  a  result  of  the  Commission's  findings,  Congress  passed 
the  DoD  Reorganization  Act  of  1986.  Although  it  recommended 
sweeping  changes,  some  requiring  Congressional  action,  the  act 
failed  to  implement  major  recommendations.  Consequently, 
President  Bush  directed  a  1989  study.  This  study,  called  the 
Defense  Management  Review,  led  to  extensive  changes  to  the  DoD 
acquisition  organization.  It  streamlined  the  acquisition 
chain-of -command  from  the  Defense  Acquisition  Executive 
through  a  newly  created  Service  Acquisition  Executive  (SAE) . 
The  chain-of -command  continued  from  the  SAE  to  the  Program 
Executive  Officer  (PEO)  and  to  the  program  manager  (PM)  .  [Ref. 
3:  p.  5] 

Through  the  years  in  addition  to  executive  level 
scrutiny,  Congress  has  taken  an  increased  degree  of  interest 
in  DoD  acquisition  efforts.  Especially  since  1970,  Congress 
has  accelerated  legislation,  enacting  implementing  regulations 
that  further  adds  to  the  government  bureaucracy  and  further 
complicates  the  acquisition  process . [Ref .  3:  p.  6] 

For  example,  a  goal  of  Congress  in  the  1970s  and  1980s 
was  to  implement  full  and  open  competition  in  defense 
acquisition.  Several  laws  within  the  last  two  decades 
underscore  the  importance  of  achieving  that  goal .  These 
actions  include:  The  1978  Acquisition  and  Distribution  of 
Commercial  Products  Program,  the  1984  Competition  in 
Contracting  Act,  and  the  1989  Congressional  Direction  to 
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Streamline  Regulations  Governing  Commercial  Products . [Ref .  4: 

p.  20] 

Congressional  involvement  continued  as  a  result  of  the 
Ill  Winds  procurement  scandal  in  the  1980s.  The  chairmen  of 
the  Armed  Services  Committees  joined  three  former  defense 
secretaries  and  industry  executives  to  review  the  DoD 
acquisition  system  and  concluded  little  improvement  had  been 
made  over  the  years.  The  major  product  of  their  study  was  the 
introduction  of  several  bills  that  all  proposed  centralizing 
acquisition  through  the  establishment  of  an  integrated 
acquisition  system  that  would  oversee  procurement  of  all  the 
services . [Ref .  3:  p.  7] 

One  of  the  proposed  bills  was  the  creation  of  an 
acquisition  corps  in  1990.  Representative  Nicholas  Mavroules 
(D,  Massachusetts) ,  Chairman  of  the  House  Armed  Services 
Investigation  Subcommittee,  introduced  the  bill  that  was 
ultimately  passed  as  the  Defense  Acquisition  Workforce 
Improvement  Act  (DAWIA) .  This  bill,  the  object  of  this  study, 
directed  DoD  to  create  a  professional  acquisition  corps  in 
each  of  the  services. [Ref .  3:  p.  7] 

The  history  of  acquisition  reform  reveals  many  attempts 
to  improve  the  process.  Improvement  efforts  have  been  marked 
by  hearings,  executive  commissions,  legislative  commissions, 
boards,  and  interagency  task  reviews  [Ref.  2:  p.  2]  .  Many  of 
these  efforts  often  reversed  previous  changes  to  the  process, 
resulting  in  a  cyclical  affect  of  change  to  the  system.  DAWIA 
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is  a  law  that  resulted  from  these  efforts  to  improve  the 
acquisition  system.  Next,  we  will  look  further  into  the 
events  that  led  to  DAWIA. 

C.  EVENTS  LEADING  TO  DAWIA 

1.  The  Packard  Commission 

The  Packard  Commission  study  is  likely  one  of  the  most 

influential  events  that  brought  about  the  creation  of  DAWIA. 

The  major  task  of  the  Commission  was  to  evaluate  the  defense 

acquisition  system,  to  determine  how  it  might  be  improved,  and 

to  recommend  changes  that  could  lead  to  the  acquisition  of 

military  equipment  with  equal  or  greater  performance  but  at 

lower  cost  and  with  less  delay.  [Ref.  5:  p.  4] 

The  Packard  Commission  concluded  in  their  report  that  the 

defense  acquisition  system  has  basic  problems  that  must  be 

corrected.  The  report  states: 

These  problems  result  from  an  increasingly 
bureaucratic  and  over  regulated  process.  As  a 
result,  all  too  many  of  our  weapon  systems  cost  too 
much,  take  too  long  to  develop,  and  by  the  time 
they  are  fielded,  incorporate  obsolete 
technology . [Ref .  5:  p.  5] 

The  Commission  discovered  that  the  problems  seldom  result 
from  fraud  or  dishonesty.  The  problems  were  symptomatic  of 
other  underlying  problems  affecting  the  entire  acquisition 
system.  These  problems  are  [Ref.  5:  p.  6]: 

1)  Goldplating  -  the  inclusion  of  system  features  that 
are  desired  but  not  necessary,  resulting  in  costs  that  far 
exceed  the  system's  real  value.  Goldplating  results  from 
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“user  pull”  and  “technology  push.”  User  pull  occurs  when  users 
specify  military  needs.  Frequently,  these  users  have  no 
knowledge  of  the  cost  and  schedule  implications  for  satisfying 
those  needs  and  thus  lack  any  incentive  to  compromise. 
Technology  push  occurs  when  government  or  industry  conceive  of 
new  or  advanced  technologies  and  persuade  users  to  state 
requirements  that  make  use  of  the  new  technology. 

2)  The  environment  -  The  highly  competitive  environment 
for  government  funds  forces  marketing  of  programs  in  order  to 
obtain  and  hold  on  to  funds.  When  a  program  finally  receives 
budget  approval,  the  environment  has  frequently  forced 
programs  to  overstate  requirements  and  understate  costs . 

3)  Source  Selection  -  The  environment  in  which  program 
competition  takes  place  forces  bidders  to  comply  and  make 
improvements  within  program  specifications,  not  to  develop 
modifications  that  deviate  from  these  specifications.  This 
results  in  less  emphasis  on  performance  and  more  on  optimizing 
costs.  Frequently  the  bid  goes  to  the  contractor  with  the 
most  optimistic  bid.  Optimistic  bids  result  in  understated 
costs  or  underbidding.  The  underbidding  contractor  will  try 
to  recover  by  negotiating  performance  tradeoffs  or  submitting 
engineering  change  orders . 

4)  PM  priorities  -  Instead  of  managing  the  program,  the 
PM  spends  a  great  deal  of  time  answering  to  an  army  of  special 
interest  advocates.  Each  of  these  advocates  can  demand  that 
the  program  manager  take  or  refrain  from  taking  some  action, 
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but  none  of  them  has  any  responsibility  for  the  ultimate  cost, 
schedule,  or  performance  of  the  program.  These  special 
interests  include  DoD  agencies,  contractors,  Congress,  and  a 
host  of  special  interest  groups.  All  of  these  pressures  force 
the  program  manager  to  spend  most  of  his  time  briefing  his 
program  as  opposed  to  managing  it .  The  report  states : 

The  above  problems  result  in  an  extremely  long 
acquisition  cycle  -  typically  ten  to  fifteen  years  for  major 
programs.  Long  acquisition  cycles  lead  to  further  problems 
[Ref .  5 :  p .  8]  : 

-  High  development  costs .  The  longer  it  takes  to  develop 
and  field  a  system,  the  more  expensive  the  process. 

-  Obsolete  technology.  By  the  time  the  system  is 
fielded,  its  technology  is  possibly  obsolete. 

-  Goldplating.  In  anticipation  of  long  lead  times  for 
fielding  the  system,  users  may  be  inclined  to  overstate  the 
requirements  and  the  threat  in  order  to  counter  obsolescence. 

None  of  the  above  problems  directly  point  to  the 
acquisition  workforce  and  the  need  to  improve  the  quality  of 
the  acquisition  workforce.  They  point  to  factors  external  to 
the  workforce  -  the  process,  user,  PM  demands,  etc.  To 
determine  the  link  with  the  workforce,  we  must  look  further 
into  the  Packard  Commission  Report  to  a  study  the  report 
cited.  This  study  was  undertaken  by  the  Defense  Science  Board 
(DSB) [Ref.  5:  p.  11] .  The  DSB  compared  typical  DoD 
development  programs  with  successful  programs  from  private 
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industry  and  discovered  six  underlying  features  common  to 
these  successful  commercial  programs . 

One  feature  was  the  small,  high  quality  staffs  of  these 
commercial  programs . 

Generally,  commercial  program  management 
staffs  are  much  smaller  than  in  typical  defense 
programs,  but  personnel  are  hand- selected  by  the 
program  manager  and  are  of  very  high  quality. 
Program  staffs  spend  their  time  managing  the 
program,  not  selling  it  or  defending  it.  [Ref.  5:  p. 

12] 

The  above  observation  by  the  DSB  resulted  in  the  Packard 
Commission's  seventh  recommendation:  Enhance  the  Quality  of 
Acquisition  Personnel.  It  states: 

DoD  must  be  able  to  attract  and  retain  the  caliber 
of  people  necessary  for  a  quality  acquisition  program. 
Significant  improvements  should  be  made  in  the  senior- 
level  appointment  system.  The  Secretary  of  Defense 
should  have  increased  authority  to  establish  flexible 
personnel  management  policies  necessary  to  improve 
defense  acquisition.  An  alternate  personnel  management 
system  should  be  established  to  include  senior 
acquisition  personnel  and  contracting  officers  as  well  as 
scientists  and  engineers.  Federal  regulations  should 
establish  business-related  education  and  experience 
criteria  for  civilian  contracting  personnel,  which  will 
provide  a  basis  for  the  professionalization  of  their 
career  paths.  Federal  law  should  permit  expanded 
opportunities  for  the  education  and  training  of  all 
civilian  acquisition  personnel  [Ref.  5:  p.  27] . 

With  this  recommendation,  the  Packard  Commission  linked 

the  performance  of  the  acquisition  system  to  management  of 

acquisition  personnel.  They  determined  that  progress  in  the 

performance  of  the  acquisition  system  demands  improvement  in 

the  management  of  acquisition  personnel. 

The  report  compared  the  acquisition  workforce  with  its 
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civilian  counterparts  and  concluded,  “this  work  force  is 
undertrained,  underpaid,  and  inexperienced .”  [Ref .  5:  p.  28] 
The  report  stressed  the  importance  of  enhancing  the  quality  of 
the  defense  acquisition  work  force- -both  by  attracting 
qualified  new  personnel  and  by  improving  the  training  and 
motivation  of  current  personnel. 

In  summary,  the  Packard  Commission  found  many  problems 
with  the  acquisition  process.  None  of  these  problems  seemed 
to  directly  result  from  the  quality  of  the  acquisition 
workforce.  However,  as  a  result  of  the  DSB  study  that 
compared  the  acquisition  workforce  to  its  industry 
counterparts,  the  Packard  Commission  recommended  improving  the 
quality  of  the  acquisition  workforce. 

2.  Congressional  Hearings 

The  House  Armed  Services  Committee,  Investigations 
Subcommittee,  held  hearings  regarding  the  acquisition 
workforce  on  28  March  and  24  April  1990.  These  hearings 
ultimately  led  to  the  passing  of  DAWIA.  Witnesses  at  these 
hearings  were  leaders  in  DOD  acquisition,  academia,  and 
industry.  They  included  current  and  former  Under  Secretaries 
of  Defense,  the  CEO  of  Martin-Marietta,  the  chairman  of  the 
President's  Blue  Ribbon  Commission  on  Defense  Management,  and 
the  Commandant  of  the  Defense  Systems  Management  College. 

The  subcommittee  conducted  hearings  on  proposals  for 
creating  a  “rationally  and  logically  structured  acquisition 
workforce  which  would  serve  as  the  foundation  for  a  high 
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quality  professionalized  acquisition  corps  of  senior 
executives."  [Ref .  6:  p.  1]  The  chairman  of  the  Investigations 
Subcommittee,  Hon.  Nicholas  Mavroules  stated: 

For  many  years,  now,  we  have  enacted  all  sorts  of 
legislation  dubbed  'Acquisition  Reform' .  We  have  changed 
the  process.  We  have  changed  the  procedures.  We  have 
changed  many  of  the  roles.  But  we  have  not  yet  addressed 
the  most  important  element  in  the  equation:  people. 

For  the  past  year-and-a-half ,  the  staff  has  been 
collecting  data  on  the  quarter  million  people  who  spend 
about  $140  billion  a  year  in  taxpayer  money.  Here  are  a 
few  things  we  have  learned.  The  fragmented  training 
system  requires  12  courses  on  contracting,  but  none  for 
systems  engineering  or  logistics,  although  they  are  key 
cost  drivers.  Less  than  a  third  of  those  assigned  as 
program  managers  of  major  Navy  systems  have  ever  attended 
a  course  on  program  management . 

. . .we  need  to  pay  more  attention  to  the  people  in 
the  acquisition  field.  We  need  to  train  them  better.  We 
need  to  pay  more  attention  to  their  career  paths.  We 
need  to  prepare  them  as  professionals  and  then  we  need  to 
respect  them  as  professionals. [Ref .  6:  p.  1] 


It  was  clear  from  these  hearings  that  the  consensus  among 
committee  members  and  witnesses  was  that  the  intent  of  the 
subcommittee  proposals  was  fully  supported.  Various 
approaches  were  presented  and  witnesses  voiced  some  concerns 
over  details  of  the  proposed  legislation,  but  on  the  whole  all 
attendees  agreed  with  the  proposal  to  develop  and  manage  a 
highly  trained,  motivated,  and  professional  acquisition 
workforce . 

D.  DAWIA 

The  Defense  Acquisition  Workforce  Improvement  Act,  Public 
Law  101-510,  Title  XII  was  passed  on  November  5,  1990.  The 
law  directs  the  following  [Ref.  7:  p.  1638] : 
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-  Revises  policies  and  procedures  for  the  recruitment, 
training,  and  career  development  of  military  and  civilian 
defense  acquisition  personnel. 

-  Requires  DoD  to  establish  acquisition  corps  for  each 
military  department  and  for  other  defense  agencies. 

-  Directs  DoD  to  establish  policies  and  procedures  for 
acquisition  personnel  education  and  training  programs, 
including  intern,  cooperative  education,  and  scholarship 
programs . 

-  Establishes  a  defense  acquisition  university. 

-  Authorizes  special  pay  for  acquisition  officers  in 
certain  critical  positions,  and  repayment  of  student  loans  to 
facilitate  recruitment  or  retention  of  acquisition  personnel. 

DAWIA  comprises  five  subchapters.  These  subchapters  are 
summarized  below. 

1.  Subchapter  I,  General  Authorities  and 
Responsibilities 

Outlines  roles,  authorities,  and  responsibilities  for  the 
Secretary  of  Defense,  Under  Secretary  of  Defense  for 
Acquisition,  the  Director  of  Acquisition  Education,  Training, 
and  Career  Development,  Service  Acquisition  Executives,  and 
the  Directors  of  Acquisition  Career  Management  in  the  military 
departments.  This  subchapter  directs  the  Secretary  of  each 
military  department,  acting  through  the  service  acquisition 
executive,  to  establish  an  acquisition  career  program  board  to 
advise  the  service  acquisition  executive  in  managing  the 
accession,  training,  education,  and  career  development  of 
military  and  civilian  personnel  in  the  acquisition  workforce 
and  in  selecting  individuals  for  an  Acquisition  Corps.  [Ref.  7: 
p.  1638] 

2.  Subchapter  IX,  Defense  Acquisition  Positions 

Directs  the  Secretary  of  Defense  to  designate  those 
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positions  in  the  Department  of  Defense  that  are  acquisition 
positions.  The  acquisition  positions  must  include  the 
following  areas  [Ref.  7:  p.  1640]: 

-  Program  management 

-  Systems  planning,  research,  development,  engineering, 
and  testing 

-  Procurement,  including  contracting 

-  Industrial  property  management 

-  Logistics 

-  Quality  control  and  assurance 

-  Manufacturing  and  production 

-  Business,  cost  estimating,  financial  management,  and 
auditing 

-  Education,  training,  and  career  development, 

-  Construction 

-  Joint  development  and  production  with  other  government 
agencies  and  foreign  countries 

Section  1722  directs  the  Secretary  of  Defense,  acting 

through  the  DAE,  to: 

. . . ensure  that  appropriate  career  paths  for 
civilian  and  military  personnel  who  wish  to  pursue 
careers  in  acquisition  are  identified  in  terms  of 
the  education,  training,  experience, _  and 
assignments  necessary  for  career  progression  of 
civilians  and  members  of  the  armed  forces  to  the 
most  senior  acquisition  positions.  The  Secretary 
shall  make  available  published  information  on  such 
career  paths. [Ref.  7:  p.  1641] 

This  section  further  directs  the  Secretary  to  limit 
preference  for  military  personnel  while  taking  actions  to 
ensure  that  civilians  are  given  maximum  opportunity  to  qualify 
for  senior  acquisition  positions  through  selection  of  the  best 
qualified  individuals.  It  also  directs  the  Secretary  to 
increase  the  proportion  of  civilians  serving  in  critical 
acquisition  positions . [Ref .  7:  p.  1641] 

This  law  directs  the  Secretary  of  Defense  to  establish 
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education,  training,  and  experience  requirements  for  each 
acquisition  position.  Section  1724  specifically  outlines 
qualification  requirements  for  contracting  positions .  [Ref .  7: 
p.  1642] 

3.  Subchapter  III,  Acquisition  Corps 

Directs  the  Secretary  of  Defense  to  establish  an 
Acquisition  Corps  for  each  of  the  military  departments.  The 
SECDEF  must  ensure  that  the  officers  within  the  Acquisition 
Corps  are  promoted  at  the  same  rate  as  all  line  officers. 
Selection  and  eligibility  criteria  must  be  established 
regarding  education  and  experience .  [Ref .  7:  p.  1644] 

Acquisition  Corps  qualification  criteria  are  specified  in 
this  subchapter.  For  example,  it  requires  corps  members  to 
have  a  baccalaureate  degree  at  an  accredited  educational 
institution  and  at  least  24  semester  credit  hours  (or  the 
equivalent)  of  study  from  an  accredited  institution  of  higher 
education  from  among  the  following  disciplines:  accounting, 
business  finance,  law,  contracts,  purchasing,  economics, 
industrial  management,  marketing,  quantitative  methods,  and 
organization  and  management  or  24  semester  credit  hours  (or 
the  equivalent)  from  an  accredited  institution  of  higher 
education  in  the  person's  career  field  and  12  semester  credit 
hours  from  such  an  institution  from  among  the  above 
disciplines . [Ref .  7:  p.  1645] 

According  to  this  subchapter,  critical  acquisition 
positions  can  be  filled  only  by  members  of  the  Acquisition 
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Corps .  The  SECDEF  must  designate  the  acquisition  positions 
that  are  critical  acquisition  positions.  The  positions  must 
include  acquisition  positions  requiring  civilian  grades  of  GS- 
14  or  above,  Senior  Executive  Service,  military  members  in  the 
grade  of  0-5  or  above,  program  executive  officer  positions, 
program  managers,  and  “any  other  acquisition  position  of 
significant  responsibility  .  .  .  "[Ref.  7:  p.  1643] 

Subchapter  III  also  specifies  assignment  periods  for 
program  managers,  stating: 

. . .a  program  manager  and  a  deputy  program  manager  of 
a  major  defense  acquisition  program  must  be  assigned  to 
the  position  at  least  until  completion  of  the  major 
milestone  that  occurs  closest  in  time  to  the  date  on 
which  the  person  has  served  in  the  position  for  four 
years  .  .  .  [Ref.  7:  p.  1647] 

Subchapter  III  further  specifies  experience  and  education 
requirements  for  program  managers,  deputy  program  managers, 
program  executive  officers,  general  officers  and  civilian 
equivalent  positions,  and  senior  contracting  officials.  For 
example,  prior  to  being  assigned  to  a  program  manager 
position,  a  person  must  have  completed  the  program  management 
course  at  Defense  Systems  Management  College  or  comparable 
program  (with  SECDEF  approval),  and,  for  major  programs,  have 
at  least  eight  years  of  experience  in  acquisition,  at  least 
two  years  of  which  were  performed  in  a  program  office  or 
similar  organization. [Ref .  7:  p.  1649] 

4.  Subchapter  IV,  Education  and  Training 

Directs  the  SECDEF  to  establish  policies  and  procedures 
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for  establishing  and  implementing  education  and  training 
programs.  Programs  to  be  established  include  intern  programs, 
cooperative  education  programs,  scholarship  programs,  and 
others  such  as  tuition  reimbursement  and  student  loan 
repayment.  The  SECDEF  is  also  required  to  establish  and 
maintain  a  defense  acquisition  university  structure  to  educate 
the  acquisition  workforce  and  provide  research  and  analysis  of 
defense  acquisition  policy  issues  from  an  academic 
perspective . [Ref .  7:  p.  1651] 

5.  Subchapter  V,  General  Management  Provisions 

Prescribes  general  management  requirements  and 
authorities.  The  chapter  specifically  directs  the  military 
departments  and  Defense  Agencies  to  establish  a  management 
information  system  capable  of  providing  standardized 
information  to  the  Secretary  on  persons  serving  in  acquisition 
positions .  [Ref .  7:  p.  1653] 

With  the  passing  of  DAWIA,  we  have  a  law  that  prescribes 
the  management  policies  for  the  acquisition  workforce.  This 
law  outlines  qualification  requirements,  assignment  policies, 
career  progression,  and  overall  policies  and  structure  for 
managing  the  acquisition  workforce. 


E .  CHAPTER  SUMMARY 

Throughout  U.S.  history,  the  Defense  acquisition  process 
has  been  subjected  to  a  great  deal  of  reform  efforts,  largely 
as  a  result  of  negative  perceptions  about  its  efficiency  as  an 
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acquisition  system.  These  perceptions  emanate  from  executive 
administrations,  Congress,  and  the  public.  Numerous 
acquisition  reform  initiatives  have  tried  to  alleviate  these 
perceptions.  These  reform  initiatives  have  historically  been 
cyclical  and  are  largely  implemented  without  provisions 
requiring  the  measurement  of  their  intended  effect  on  the 
acquisition  process. 

DAWIA  is  an  example  of  such  a  reform  initiative.  DAWIA's 
objectives  are  primarily  to  improve  the  quality  of  the 
acquisition  workforce  and  to  improve  the  acquisition  process. 
How  do  we  determine  whether  these  objectives  are  being 
achieved?  Metrics  could  be  the  answer.  The  next  chapter 
introduces  metrics  and  explores  their  use. 
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Ill .  METRICS 


A.  INTRODUCTION 

This  chapter  introduces  metrics,  providing  definition, 
purpose,  and  benefits.  The  chapter  describes  a  methodology 
for  establishing  metrics  in  an  organization.  The  chapter 
then  presents  metrics  used  in  other  fields  (e.g.  software, 
education) and  presents  metrics  proposed  within  DoD  for 
acquisition  reform. 

B.  METRICS  DEFINED 

In  most  technical  endeavors,  measurement  and  metrics  help 
us  to  understand  the  technical  process  that  is  used  to  develop 
a  product.  The  process  is  measured  in  an  effort  to  improve 
it . [Ref.  8 :  p.  43] 

At  first,  it  would  appear  that  using  measurement  in 
processes  is  a  highly  desirable  tool.  After  all,  measurement 
enables  us  to  quantify  and  therefore  manage  more  effectively. 
But  reality  can  be  somewhat  different.  Choosing  the 
appropriate  measures  can  be  quite  difficult  and  the  choice  can 
invite  a  lot  of  argument  and  controversy.  What  are 
appropriate  metrics  for  the  process?  Is  it  fair  to  use 
measurements  to  compare  people,  processes,  or  products?  These 
questions  and  dozens  of  others  always  surface  when  an  attempt 
is  made  to  measure  something  that  has  not  been  measured  in  the 
past. [Ref.  8:  p.  43] 
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Lord  Kelvin  once  said: 


When  you  can  measure  what  you  are  speaking 
about  and  express  it  in  numbers,  you  know  something 
about  it;  but  when  you  cannot  measure,  when  you 
cannot  express  it  in  numbers,  your  knowledge  is  of 
a  meager  and  unsatisfactory  kind:  it  may  be  the 
beginning  of  knowledge,  but  you  have  scarcely,  in 
your  thoughts,  advanced  to  the  stage  of  a 
science . [Ref .  8:  p.  45] 


C.  MEASURING  IN  ORGANIZATIONS 

An  organization  is  a  system  with  three  levels  of 
performance- -organization,  process,  and  job/performer- -that 
must  be  managed  in  order  to  get  consistent,  high-level 
organization  output.  We  measure  within  an  organization  so 
that  we  can  monitor,  control,  and  improve  system  performance 
at  all  three  levels.  We  measure  in  order  to  [Ref.  9:  p.  141] : 

-  Specifically  communicate  performance  expectations  to 
subordinates 

-  Know  what  is  going  on  in  the  organization 

-  Identify  performance  gaps  that  should  be  analyzed  and 
eliminated 

-  Provide  feedback  that  compares  performance  to  a 
standard 

-  Identify  performance  that  should  be  rewarded 

-  Effectively  make  and  support  decisions  regarding 
resources,  plans,  policies,  schedules,  and  structure 

-  Identify  areas  for  improvement 

Another  metrics  perspective  was  presented  by  the  Process 
Action  Team  (PAT)  for  contract  administration  reform  in 
February  1995.  The  PAT  identifies  metrics  that  simply  record 
whether  an  event  has  happened  or  not  (go/no-go  metrics) , 
metrics  that  determine  the  extent  to  which  an  action  is  taking 
place  (activity  metrics) ,  and  metrics  that  measure  whether 
actions  are  achieving  the  desired  results . [Ref .  10:  p.  37] 


Effective  use  of  measures  means  not  only  establishing 
measures,  but  using  them  properly.  Measures  must  be  used  to 
monitor  the  things  in  an  organization  that  are  linked  to 
organization  goals  and  objectives.  The  measurement  system,  in 
its  totality,  should  contain  related  measures  that  are  not 
counterproductive.  Finally,  the  measurement  data  must  be 
collected  and  used  for  the  benefit  of  the  organization  through 
intelligent  action. [Ref.  9:  p.  142] 

Rummler  and  Brache,  in  “Improving  Performance”  developed 
the  following  sequence  for  establishing  measures  in  an 
organization  [Ref.  9:  p.  143] : 

1.  Identify  the  most  significant  outputs  of  the 
organization,  process,  or  job. 

2.  Identify  the  “critical  dimensions”  of  performance  for 
each  of  these  outputs.  Critical  dimensions  of  quality  include 
accuracy,  ease  of  use,  novelty,  reliability,  ease  of  repair, 
and  appearance.  Critical  dimensions  should  be  derived  from 
the  needs  of  the  internal  and  external  customers  who  receive 
the  outputs  and  from  the  financial  needs  of  the  business. 

3.  Develop  the  measures  for  each  critical  dimension. 
For  example,  if  “ease  of  use"  is  a  critical  dimension  of 
quality  for  a  given  output,  one  or  more  measures  should  answer 
this  question:  “What  indicators  will  tell  us  if  our  customer 
finds  our  product  or  service  (output)  easy  to  use?” 
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D.  MEASURING  IN  VARIOUS  FIELDS 

Metrics  exist  in  many  fields,  and  each  field  approaches 
the  use  of  metrics  in  a  unique  way.  The  fields  presented  are 
software  development  and  training/education.  These  fields 
were  chosen  because  their  products  are  difficult  to  measure 
and  thus  present  similar  metrics  challenges  to  those  facing 
acquisition  reform  and  DAWIA.  Each  field  has  attempted  to 
face  the  challenge  of  developing  metrics  that  accurately  and 
effectively  measure  what  is  actually  intended  to  be  measured. 

1.  Software 

Software  metrics  refer  to  a  broad  range  of  measures  for 
computer  software.  The  primary  focus  of  these  measures  is 
with  productivity  and  quality- -measures  of  output  and  fitness 
of  use.  The  software  community  is  interested  in  these  types 
of  metrics  for  planning  and  estimation  purposes.  They  want  to 
be  able  to  determine  productivity  and  quality  on  past  projects 
and  determine  how  the  productivity  and  quality  data  can  be 
extrapolated  to  the  present. [Ref.  8:  p.  45] 

We  measure  software  for  many  reasons:  (1)  to  indicate  the 
quality  of  the  product;  (2)  to  assess  the  productivity  of  the 
people  who  produce  the  product;  (3)  to  assess  the  benefits 
derived  from  new  methods  and  tools;  (4)  to  form  a  baseline  for 
estimation;  (5)  to  help  justify  requests  for  new  tools  or 
additional  training . [Ref .  8:  p.  45] 

The  software  community  categorizes  metrics  in  two  ways: 
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direct  and  indirect  measures .  Direct  measures  are  often 
physical,  tangible  measurements  of  a  process  or  product  such 
as  cost,  speed,  size,  and  number  of  defects.  Indirect 
measures,  less  straightforward  measurements,  include 
functionality,  quality,  complexity,  efficiency,  reliability, 
maintainability,  and  many  other  “abilities [Ref .  8:  p.  46] 

Software  metrics  have  been  further  categorized  into  the 
following:  Productivity  metrics  focus  on  output,  quality 
metrics  provide  an  indication  of  conformance  to  a  standard, 
technical  metrics  focus  on  the  degree  of  complexity  of  a 
process  or  product,  size-oriented  metrics  are  used  for  direct 
measures,  function- oriented  metrics  provide  indirect  measures, 
and  human-oriented  metrics  collect  information  about  the 
manner  in  which  people  develop  products  and  the  human 
perception  about  the  effectiveness  of  tools  and  methods .  [Ref . 
8.  p.  46] 

2 .  Training  and  Education 

Many  organizations  expend  a  great  deal  of  resources  on 
training  and  education  programs  for  its  employees,  and  the 
government  is  certainly  no  exception.  These  organizations 
assume  or  hope  that  these  programs  provide  benefit  to  the 
organization  with  the  ultimate  objective  of  improving  the 
performance  of  the  organization. 

Should  the  benefits  of  the  training  and  education  be 
based  on  assumptions  and  hope  or  should  actions  be  taken  to 
measure  the  effectiveness  of  the  training  and  education?  Most 
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organizations  would  prefer  the  latter  for  two  reasons:  1)  to 
justify  the  expenditure  of  resources  on  training  and 
education; and  2)  to  improve  the  training  and  education 
programs  in  order  to  better  meet  the  needs  of  the 
organization.  This  section  explores  ways  to  measure  the 
effectiveness  of  training  and  education. 

The  key  to  measuring  the  effectiveness  of  training  and 
education  is  to  begin  with  the  overall  organization 
objectives.  After  overall  organization  objectives  have  been 
identified,  the  following  steps  should  be  taken  [Ref.  11:  p. 
180]  : 

1)  Identify  employee  performance  behaviors  that  enable 
the  organization  to  accomplish  organization  objectives. 

2)  Establish  training  to  develop  those  performance 
behaviors . 

3)  Measure  the  resulting  performance  behavior  against  the 
desired  performance  behavior  to  determine  the  effect  of  the 
training . 

4)  Measure  improvement  in  organization  performance. 

A  training  program  should  ultimately  exist  to  change  an 
old  behavior  to  a  new  behavior.  This  new  behavior  should  have 
some  type  of  effect  on  the  organization.  If  the  new  behavior 
is  a  desired  behavior  linked  to  organizational  objectives, 
then  the  effect  should  be  positive . [Ref .  11:  p.  181] 

However,  linking  training  to  organizational  performance 
is  likely  to  be  very  difficult.  Organizational  performance 
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improvements  could  result  from  many  factors  other  than  from 
training  unless  there  is  certainty  that  no  other  factors 
exist .  Factors  could  include  other  areas  such  as  process 
improvements,  better  products,  organizational  changes,  etc. 
Therefore,  a  more  effective  evaluation  of  training  programs 
would  be  limited  to  steps  1,  2,  and  3  above.  Odiorne  states: 

If  a  course  defines  its  intentions  as  changing 
specific  old  behavior  to  specific  new  behavior  and 
this  change  actually  occurs,  the  training  must  be 
considered  successful.  The  evaluation  of  training 
is  limited  to  assessing  or  measuring  as  accurately 
as  possible  how  much  of  the  desired  (objective) 
behavior  was  actually  attained  and  applied:  first, 
in  the  class,  and  second,  back  on  the  job. 

If  training  actually  changed  behavior  in  the 
class  then  it  must  be  considered  a  training  success 
from  the  class  standpoint  and  instruction  technique 
standpoint.  If  it  fails  to  convert  back  to  the 
job,  then  the  analysis  of  system  support  of  that 
behavior  may  be  at  fault  and  not  the  training,  but 
the  prior  planning  and  task  analysis  are  probably 
at  fault. [Ref.  11:  p.  182] 

Training  and  education  programs  consume  a  great  deal  of 
organization  resources- -time,  people,  and  money- -resources 
that  could  be  used  elsewhere  to  benefit  the  organization. 
Cost  effectiveness  studies  aim  to  determine  the  benefit 
received  from  training  and  education  programs  compared  to  the 
resource  cost  for  these  programs . [Ref .  11:  p.  183] 

Cost  effectiveness  studies  of  training  require  that 
objectives  for  training  be  set  and  that  these  objectives  have 
the  following  characteristics  [Ref.  11:  p.  190] : 

1)  They  define  the  behavior  change  sought,  stating  what 
the  present  behavior  is  and  its  consequences  and  the  desired 
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behavior  and  its  consequences. 

2)  These  must  be  stated  in  mixtures  of  words  and  numbers, 
which  calls  for  quantifying  certain  key  parts  of  the 
behavioral  objectives  in  order  that  they  might  be  used  later 
as  criteria  for  measuring  contribution  of  the  training  effort. 

However,  measuring  the  effect  of  training  and  education 
can  be  quite  difficult.  Many  aspects  of  human  behavior  do  not 
lend  themselves  to  quantification.  The  reliability  of 
measures  varies.  Odiornes  identifies  the  following  measures 
in  descending  order  of  reliability  [Ref.  11:  p.  191]: 

1)  Hard,  raw,  real  time  data  comprise  the  best  kind  of 
measurement.  Dollars  of  expense  cut,  reduced  performance 
time,  and  quantity  of  items  produced  are  examples.  This  type 
of  data  is  objective  and  can  be  easily  verified. 
Unfortunately,  it  can  be  very  difficult  to  directly  tie  this 
type  of  data  to  training. 

2)  Indices  and  percentages  are  used  to  denote  change, 
including  changed  behavior.  This  measurement  tool  is  less 
reliable  than  hard  data,  but  nonetheless  can  be  quite 
valuable.  Batting  coaches  will  watch  the  change  in  their 
students'  batting  average,  slugging  percentage,  and  runs 
batted  in  percentage  to  evaluate  the  effect  of  their  teaching. 

3)  Position  on  a  scale  is  a  third  type  of  measurement 
device  where  subjects  use  ratings  to  make  assessments.  On  a 
scale  of  one  to  five,  a  supervisor  may  be  asked  to  rate  his 
employee's  performance  prior  to  training  and  after  training. 


32 


4)  Verbal  descriptions,  the  least  reliable  and  most 
subjective,  are  used  when  no  other  measure  is  possible. 

Training  evaluations  frequently  rely  on  student  opinions. 
Surveys  are  gathered  at  the  completion  of  the  class  or  at  the 
completion  of  each  individual  session.  These  surveys  provide 
an  assessment  on  the  quality  of  instruction  but  provide  no 
input  on  how  the  instruction  impacts  work  performance.  [Ref. 
11:  p.  192] 

Opinions  on  training  can  also  be  obtained  from  third 
parties  such  as  employee  supervisors.  Supervisors  could  get 
information  from  students  about  the  class,  either  directly  or 
indirectly,  and  make  an  assessment.  This  approach  can  be  very 
disorganized,  subjective,  and  as  a  result,  unreliable. [Ref . 
11:  p.  193] 

Opinions  on  training  can  be  gathered  from  the  trainer  or 
training  administrator.  If  the  training  is  pleasing  to  these 
people,  it  may  be  considered  successful.  Clearly  this 
approach  can  result  in  misleading  assessments  when  the  needs 
of  the  trainer  or  administrator  take  precedence  over  the  needs 
of  the  student. [Ref.  11:  p.  193] 

There  are  additional,  but  less  reliable  ways  to  evaluate 
training  and  education  programs.  Participation  measures  are 
used  to  measure  participation  in  terms  of  the  number  or 
percentage  of  personnel  enrolled  or  the  percentage  of 
successful  completions.  If  this  type  of  evaluation  equates 
enrollment  or  completion  with  success  then  the  evaluation 
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fails.  Participation  is  no  indication  of  the  success  of  a 
program.  Comparison  measures  are  used  to  compare  training 
programs  to  training  programs  within  similar  organizations  or 
compare  to  a  standard  quality  program.  This  type  of 
evaluation  could  lead  to  unfortunate  conclusions  because  no 
two  organizations  are  exactly  alike  and  a  standard  quality 
program  is  based  upon  someone's  subjective  determination  of 
what  a  quality  program  should  be. [Ref.  12:  p.  20] 

There  are  many  ways  to  measure  and  evaluate  the  success 
of  training  and  education  programs.  Measures  and  evaluation 
based  upon  measurable  criteria  that  relate  to  organization 
objectives  are  the  most  feasible  and  fruitful  [Ref.  12:  p. 
21]  . 


E.  ACQUISITION  REFORM  METRICS 

The  focus  on  acquisition  reform  (AR)  in  recent  years  has 
resulted  in  increased  efforts  within  DoD  to  develop  metrics 
for  assessing  the  efficiency  gains  within  DoD  acquisition. 
The  Assistant  Secretary  of  the  Army  for  Research,  Development, 
and  Acquisition,  Mr.  Gilbert  Decker,  stated  the  following  as 
one  of  his  goals  in  his  Acquisition  Reform  Strategic  Plan  for 
all  Army  acquisition  organizations  [Ref.  13] : 

Put  metrics  in  place  to  measure  progress . 
Elaborate  reporting  and  feedback  systems  are 
counterproductive,  but  a  few  key,  value-added 
metrics  are  necessary  to  focus  efforts  and 
determine  progress.  AR  metrics  should  be  developed 
to  support  the  command  strategy,  be  directly 
related  to  measurement  of  outcomes,  and  measure 
progress  in  the  following  high  payoff  areas: 
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1)  Use  metrics  to  measure  the  rapidity  of 
technology  insertion  into  fielded  systems. 

2)  Measure  increased  use  of  performance 
specifications 

3)  Measure  the  reduction  in  acquisition  cycle 
and  procurement  lead-times. 

4)  Measure  total  cost  reductions. 

1.  DoD  Acquisition  Reform  Metrics 

DoD  established  the  Acquisition  Reform  Benchmarking  Group 
(ARBG)  in  1996  to  develop  metrics  for  assessing  the  efficiency 
gains  within  DoD  acquisition.  These  metrics  are  in  the  early 
stages  of  development .  The  ARBG  states  “ .  .  .  data  presented 

for  each  metric  is  subject  to  refinement  as  the  metrics  are 
evaluated.  Furthermore,  metrics  may  be  further  developed  or 
discarded  in  the  future  if  deemed  appropriate  by  the 
ARBG.” [Ref.  14] 

The  ARBG  has  developed  three  levels  of  metrics; 
enterprise,  subordinate,  and  program  metrics  [Ref.  15]  . 
Enterprise  metrics  are  used  to  assess  efficiency  gains  of  the 
total  acquisition  process .  These  metrics  are  further  broken 
down  into  cost  metrics,  schedule  metrics,  training  metrics, 
and  performance  metrics.  Subordinate  metrics  are  the  next 
level  of  metrics.  These  metrics  measure  the  subordinate  tasks 
or  elements  that  contribute  to  enterprise  metrics.  Program 
metrics  are  the  lowest  level  metrics  developed  by  the  ARBG. 
These  metrics  measure  factors  relative  to  specific  acquisition 
programs . 

Following  are  descriptions  and  diagrams  of  enterprise, 
subordinate,  and  program  metrics: 
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a.  Enterprise  Metrics 

(1)  '  Annual  Rate  Change.  A  cost  metric 
intended  to  capture  enhanced  acquisition  process  control  due 
to  the  implementation  of  Integrated  Product  Teams  (IPTs)  and 
Cost  as  an  Independent  Variable  (CAIV)  (Figure  1) .  The  use  of 
IPTs  and  CAIV  is  expected  to  diminish  the  historic  fluctuation 
in  program  cost  growth.  This  metric  is  calculated  using 
Selected  Acquisition  Report  (SAR)  data  from  common  DoD  major 
defense  acquisition  programs  across  sequential  years. [Ref.  16] 
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Figure  1.  From  Ref  [16] 


(2)  Purchasing  Cost.  A  cost  metric  that 
presents  the  in-house  costs  incurred  to  purchase  one  dollar  of 
goods  and  services,  as  a  measure  of  in-house  .ef  f  iciencies 
(Figure  2) .  Simplified  acquisition  procedures  and  procurement 
process  improvements  are  expected  to  decrease  purchasing  costs 
over  time.  The  metric  uses  personnel  data  translated  into 
purchasing  cost  by  using  salary  and  fringe  benefits  data  from 
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the  Service  personnel  offices,  OSD,  and  DoD  agencies.  The 
purchasing  dollar  value  is  calculated  based  on  procurement 
contract  awards. [Ref.  17] 


Government  Purchasing  Cost 
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Figure  2.  From  Ref.  [17] 

(3)  Cycle  Time.  A  schedule  metric  that 
measures  DoD  progress  in  streamlining  acquisition  decision 
processes  and  procedures  to  reduce  the  amount  of  time  between 
milestones  from  the  concept  definition  through  production  and 
deployment  of  a  weapon  and/or  information  system  for  major 
defense  acquisition  programs  (MDAP)  (Figure  3) . [Ref.  18] 

(4)  On  Time  Delivery.  A  schedule  metric 
intended  to  capture  the  effects  of  IPTs  and  enhanced  selection 
processes  (such  as  past  performance)  (Figure  4) .  The  metric 
presents  the  percent  of  contract  line  items  which  are  on 
schedule  in  accordance  with  their  contract  terms.  Data 
includes  line  items  from  major  weapons  systems  to  consumables. 
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Figure  4.  From  Ref.[19] 
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(5)  Workforce  Development.  A  training  metric 
that  presents  DoD’s  progress  *in  continuing  to  develop  the 
acquisition  workforce  through  training  (Figure  5) .  The  metric 
measures  the  percentage  of  certified  personnel  across  the 
number  of  coded  positions  requiring  certification. [Ref .  20] 
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Figure  5.  From  Ref.  [20] 

(6)  Quality.  A  performance  metric  intended  to 
capture  the  effect  of  numerous  DoD  initiatives  (MIL-STD 
reform,  regulatory  reform,  IPTs)  on  the  product  and  process 
quality  of  items  procured  by  DoD,  including  quality  of  design, 
development  and  manufacture  (Figure  6)  .  [Ref.  21] 

(7)  MDAP  Breach.  A  performance  metric  that 
captures  efficiency  gains  due  to  IPTs  associated  with  program 
breaches  and  resolution  of  those  breaches  (Figure  7) .  Over 
time,  both  the  number  of  breaches  and  time  to  resolve  them  are 
expected  to  decline.  The  metric  identifies  the  total  number 


39 


of  acquisition  program  baselines  (APBs)  as  compared  to  the 
number  of  breaches  resolved  (cost,  schedule,  and  performance) 
and  the  nev;  breaches  each  month.  [Ref.  22] 
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Figure  6.  From  Ref.  [21] 
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Figure  7.  From  Ref. [22] 
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Jb.  Subordinate  Metrics 

(1)  Cost  Premiums.  This  metric  is  the 
observed  regulatory  cost  premium  paid  for  unique  government 
requirements  as  estimated  by  DCAA  based  upon  the  10 
Reinvention  Laboratories  (Figure  8)  .  The  cost  premium  metric 
is  a  measurement  of  the  acquisition  reform  cost  avoidance  and 
saving  that  can  be  achieved  from  joint  contractor/government 
efforts  to  eliminate  non-value  added  regulations . [Ref .  23] 
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Figure  8.  From  Ref. [23] 


(2)  Consumable  Item  Price  Index.  A  metric 
intended. to  capture  the  cost  avoidances  that  can  be  achieved 
by  adopting  world  class  business  processes  for  consumable 
items  (Figure  9)  .  The  metric  is  a  comparison  of  the  Fisher 
Ideal  Market  Basket  Index  (five  major  DLS  commodities)  with 
the  overall  Defense  Logistics  Agency  index  of  budgeted  product 
costs  for  a  sampling  of  more  than  100,000  items. [Ref.  24] 
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(3)  Conversion  Price.  Measures  the  cost 
avoidances  associated  with  acquiring  products  and  parts  from 
reliable  suppliers  based  upon  performance  (versus  “how  to") 
specifications  (Figure  10)  .  This  metric  represents  the 
estimated  cost  avoidance  from  converting  from  military  to 
commercial  specifications  on  selected  commodity  purchases  made 
by  the  Defense  Logistics  Agency. [Ref.  25] 


(4)  Commercial  Content.  An  indicator  of  the 
degree  of  commercial  material  purchased  by  DoD  (Figure  11) . 
This  metric  indicates  the  dollar  value  of  commercial  items 
compared  to  total  dollar  value  of  all  obligations  for 
Acquisition  Category  (ACAT)  I-III  programs  in  DoD. [Ref.  26] 
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Figure  10.  From  Ref.  [25] 


Figure  11.  From  Ref.[26] 

(5)  FACNET  Transactions.  Presents  the  number 
of  FACNET  transactions  for  all  DoD  organizations  (Figure  12) . 
This  metric  is  a  measure  of  the  implementation  of  commercial 
purchasing  practices  and  acquisition  procedures  such  as 
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(7)  Administrative  Lead  Time  (ALT)  .  A  metric 
that  presents  the  average  time  elapsed  'from  the  initial 
identification  of  a  need  to  the  contract  award  (Figure  14) . 
Improved  efficiencies  associated  with  acquisition  streamlining 
and  simplified  acquisition  procedures  are  expected  to  reduce 
ALT.  At  this  time,  the  metric  presents  the  average  lead  times 
for  consumables  and  reparables  (secondary  items)  across 
DoD. [Ref.  29] 


Administrative  Lead  Time 
Ccnsuma bies  and  Reparables 


Consumables  Reparables 


Figure  14.  From  Ref.  [29] 

(8)  Contract  Changes.  This  metric  presents 
the  number  of  Class  1  Engineering  Change  Proposals  (ECPs)  and 
major  waivers /deviations  for  all  DCMC  administered  contracts 
(Figure  15) .  This  metric  is  an  indicator  of  improved 
design/development  processes  '  based  on  streamlined  business 
practices,  reduced  regulatory  burdens,  MIL-STD  reform  and 
IPPD . [Ref.  30] 
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Contract  Changes 
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Figure  15.  From  Ref. [30] 


(9)  Contract  Protests.  This  metric  presents 
the  number  of  DoD  contractor  protests  of  awards  and  the  number 
of  protests  that  were  ruled  in  favor  of  the  contractor  (Figure 
16) .  This  metric  is  expected  to  decline  with  the 
implementation  of  solicitation  and  procurement  process 
reform. [Ref .  31] 

(10)  Commercial  Item  Contract.  An  indicator 
of  the  extent  of  commercial  item  acquisition  within  DoD 
procurement  (Figure  17) .  This  metric  is  the  ratio  of  contract 
awards  for  commercial  items  to  total  contract  awards  (number 
and  dollar  value)  .  [Ref.  32] 
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Figure  16.  From  Ref.  [31] 


Figure  17.  From  Ref.[33] 


(11)  Cost  and  Pricing  Data.  An  indicator  of 
reduction  in  unique  government  requests  for  pricing  data  and 
its  associated  proposal  preparation  cost,  government  and 
contractor  oversight,  *  and  restriction  of  the  commercial 
contracting  base  (Figure  18)  .  This  metric  presents  the 
percent  of  actions  requesting  Cost  and  Pricing  data  for 
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actions  over  $500,000  and  for  actions  under  $500, 000 . [Ref .  33] 


Figure  18.  From  Ref.  [33] 


(12)  Acquisition  Phase  Time.  This  metric 
measures  DoD  progress  in  streamlining  acquisition  decision 
processes  and  procedures  to  reduce  the  amount  of  time  between 
milestones  from  the  concept  definition  through  production  and 
deployment  of  a  weapon  and/or  information  system  (Figure 
19)  .  [Ref.  34] 

(13)  Logistic  Response  Time.  An  indicator  of 
DoD ' s  effort  to  move  toward  world  class  logistics  practices, 
including  simplified  acquisition  procedures  for  logistics 
items  such  as  scares  and  consumables  (Figure  20) .  This  metric 
presents  the  average  elapsed  time  between  customer  order  and 
customer  receipt  of  Defense  Logistics  Agency  (DLA)  items. [Ref. 
35] 
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Figure  19.  From  Ref. [34] 


Figure  20.  From  Ref.  [3  5] 


c.  Program  Metrics 

(1)  Military  Specifications  (MILSPECs) . 
Presents  percentage  reduction  of  the  number  of  MILSPECs  and 
standards  in  an  RFP  as  compared  to  a  traditional  program 
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(Figure  21) . [Ref.  36:  p.  1] 


Figure  21.  From  Ref. [36] 


(2)  Contract  Data  Requirements  List  (CDRL) . 
Presents  percentage  reduction  of  the  number  of  CDRLs  in  an  RFP 
as  compared  to  a  traditional  program  (Figure  22).  [Ref.  36:  p. 
2] 


(3)  Contract  Administration  Work  Hours.  A 
metric  presenting  the  percentage  reduction  of  Contract 
Administration  Services  (CAS)  work  hours  for  programs  using 
acquisition  reform  as  compared  to  traditional  programs  prior 
to  reform  (Figure  23).  [Ref.  36:  p.  3] 

(4)  Proposal  Preparation.  Percentage 
reduction  in  Request  For  Proposal  (RFP)  preparation  work  hours 
as  compared  to  a  traditional  program  (Figure  24 ) . [Ref . 36 :p.  4] 
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Percentage  Reduction  -  Compared  to  a  Traditional  Program 


Figure  22.  From  Ref.[36] 


Figure  23.  From  Ref.  [3  6] 
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Percentage  Reduction  -  Compared  to  a  Traditional  Program 


C-130J  CDE  FSCATT  JDAM  JPATS 


Figure  24.  From  Ref. [36] 


(5)  DCAA  Audit  Hours.  Percentage  reduction  in 
DCAA  audit  hours  as  compared  to  a  traditional  program  (Figure 
25) . [Ref.  36 :  p.  5] 

(6)  Proposal  Evaluation  Time.  Percentage 
reduction  in  proposal  evaluation  hours  as  compared  to  a 
traditional  program  (Figure  26). [Ref.  36:  p.  6] 

(7)  RFP  Pages.  Percentage  reduction  in  the 
number  of  RFP  pages  as  compared  to  a  traditional  program 
(Figure  27)  .  [Ref.  36:  p.  7] 

(8)  Contract  Costs.  Percentage  reduction  in 
contract  costs  (estimate  vs.  award)  as  compared  to  a 
traditional  program  (Figure  28). [Ref.  36:  p.  8] 
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Figure  26.  From  Ref. [36] 
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(9)  Program  Office  Staffing.  Percentage 
reduction  in  program  office  staffing  as  compared  to  a 

traditional  program  (Figure  29). [Ref.  36:  p.  9] 

(10)  Contract  Cost  Variance.  Percentage 
reduction  in  contract  cost  variance  as  compared  to  a 

traditional  program  (Figure  30). [Ref.  36:  p.  10] 
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Figure  27.  From  Ref.[36] 

(11)  Contract  Schedule  Variance.  Percentage 
reduction  in  the  contract  schedule  variance  as  compared  to  a 
traditional  program  (Figure  31).  [Ref.  36:  p.  11] 
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Figure  28.  From  Ref.  [3  6] 


Figure  29.  From  Ref.[36] 
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2 .  Navy  Acquisition  Reform  Metrics 

The  Department  of  the  Navy  is  working  to  develop  metrics 
to  gauge  the  implementation  of  Acquisition  Reform  Initiatives. 
Assistant  Secretary  of  the  Navy  for  Research,  Development,  and 
Acquisition  (ASN (RDA) )  memorandum,  “Navy  Acquisition  Metrics  - 
Progress  Reviews,"  requires  monthly  status  reviews  commencing 
27/28  June  1996  .  .  as  a  means  to  gauge  and  report 

progress  in  achieving  acquisition  reform  .  strategic 
goals. "[Ref.  37]  The  suggested  metrics  are  the  following 
[Ref.  38]  : 

-  Average  Cycle  Time  for  Issuance  of  RFPs 

-  Use  of  IPTs 

-  RFP  (ACAT  1-3)  Specifications/Standards  Waivers 

-  Use  of  FAR  Part  12  for  Commercial  Item  Acquisitions 

-  Number  of  RFPs  Allowing  Oral  Presentations 

-  Number  of  Electronic  Solicitations 

-  RFPs  Using  CAIV 

-  RFPs  Considering  Total  Ownership  Cost 

-  ACAT  1-4  RFPs  Using  Strategies  to  Minimize 
People/Training  Time  Needed  to  Operate  Systems 

-  Electronic  Data  Deliverables 

-  EDI  Usage 

-  Configuration  Management  Delegated  to  Contractor 

-  Use  of  Award  Fees 

-  IPTs  with  Multiple-site  Participation 


F .  CHAPTER  SUMMARY 

Metrics  can  be  a  valuable  tool  for  gauging 
implementation,  effectiveness,  and  success  within  an 
organization.  There  are  many  types  of  metrics,  therefore,  in 
order  to  be  effective  and  relevant,  metrics  must  be  tailored 
to  the  needs  of  the  organization. 
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Acquisition  reform  metrics  are  in  the  early  stages  of 
development.  Most  of  the  ARBG  metrics  focus  on  the 
acquisition  process,  gauging  the  improvement  in  cost, 
schedule,  and  performance.  The  Navy  metrics  are  geared  toward 
measuring  the  implementation  of  various  initiatives.  In  the 
next  chapter,  we  take  a  closer  look  at  DAWIA  and,  by  using  the 
information  in  this  chapter,  develop  metrics  for  DAWIA. 
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IV.  ANALYSIS  FOR  DAWIA  METRICS 

A.  CHAPTER  INTRODUCTION 

This  chapter  presents  an  analysis  of  DAWIA  metrics  using 
the  Policy  Effectiveness  Model  (Figure  32) .  This  model  is  a 
tool  used  to  analyze  and  develop  metrics  for  programs,  and 
through  its  application  one  can  identify  the  objectives  of 
DAWIA,  identify  the  activities  for  accomplishing  the 
objectives,  and  propose  metrics  for  measuring  the  activities. 


Policy  Effectiveness  Model 


Figure  32.  From  Ref.[l] 


B .  DAWIA  OBJECTIVES 

The  Packard  Commission  Report,  the  Congressional  hearings 
on  the  acquisition  workforce,  and  the  requirements  of  DAWIA 
points  to  the  following  objectives  for  this  detailed,  far 
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reaching  legislation: 

-  Provide  a  clear  career  track  for  the  acquisition 
workforce 

-  Educate  the  workforce 

-  Increase  the  number  of  civilian  acquisition  positions 

-  Provide  career  growth 

-  Grow  PMs 

-  Stabilize  programs  through  reduction  in  PM  turnover 

The  aim  of  the  above  objectives  is  to  achieve  the 

following  ultimate  objectives: 

-  Increase  acquisition  expertise  and  experience 

-  Control  program  costs 

-  Efficiently  acquire  parts,  supplies,  materials,  and 
programs 

The  first  seven  objectives  focus  on  improving  the  quality 
of  the  workforce.  The  last  two  objectives  focus  on  improving 
the  acquisition  process.  DAWIA  aims  to  produce  a  quality 
workforce  that  results  in  an  improved  acquisition  process. 
Now  we  must  determine  ways  to  measure  these  objectives. 

C.  ACTIVITIES  AND  METRICS  FOR  DAWIA  OBJECTIVES 

The  following  subsections  are  DAWIA  objectives,  as 
identified  above.  Within  each  subsection  are  the  activities 
for  accomplishing  the  objective  and  proposed  metrics  for 
measuring  each  activity. 

1.  Provide  a  Clear  Career  Track  for  the  Acquisition 
Workforce 

a)  Activity:  Identify  acquisition  workforce  positions  and 
define  a  career  path  that  provides  development  opportunities 
and  progression. 

Metrics:  To  measure  this  activity,  use  a  go/no-go  metric 
that  determines  whether  acquisition  positions  have  been 
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designated.  Use  another  go /no-go  metric  to  determine  whether 
the  positions  are  organized  and  show  career  progression.  We 
can  measure  by  surveying  acquisition  workforce  opinions  on 
their  satisfaction  with  the  career  track  established  for  their 
career  field. 

b)  Activity:  Identify  qualification  requirements  for 
acquisition  workforce  positions. 

Metrics :  Measure  by  determining  the  percentage  of 
workforce  positions  that  have  specified  qualification 
requirements.  Also  measure  by  determining  the  percentage  of 
workforce  members  occupying  acquisition  positions  that  meet 
the  position's  qualification  requirements. 

c)  Activity:  Establish  developmental  opportunities  (i.e. 
developmental  jobs,  training,  and  education) . 

Metrics :  Use  a  go/no-go  metric  to  determine  whether 
developmental  opportunities  have  been  identified.  Another 
metric  presents  the  ratio  of  the  number  of  acquisition 
workforce  members  to  the  number  of  acquisition  developmental 
slots.  An  additional  metric  could  be  used  to  measure  the 
progression  of  workforce  members  following  developmental 
assignments.  For  example,  we  could  measure  the  average  pay 
increase  or  grade  increase  for  workforce  members  who  completed 
developmental  assignments  and  compare  figures  to  those  who 
haven't  completed  developmental  assignments. 

d)  Activity:  Centrally  manage  acquisition  positions  and 
training  opportunities. 
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Metrics:  Use  a  go/no-go  metric  to  determine  whether  an 
organization  has  been  established  to  centrally  manage 
acquisition  positions.  A  metric  could  measure  the  percentage 
of  positions  that  have  been  filled  through  centralized 
selection.  Use  a  go/no-go  metric  to  determine  whether  an 
adequate  centralized  information  system  has  been  established 
to  enhance  the  military  departments'  ability  to  centrally 
manage  the  acquisition  workforce. 

e)  Activity:  Effectively  communicate  the  career  track 
information  to  the  workforce. 

Metrics:  Measure  by  surveying  a  sample  of  the  workforce 
regarding  knowledge  of  their  career  path,  qualification 
requirements,  and  developmental  opportunities.  A  resulting 
metric  could  present  the  percentage  of  acquisition  workforce 
members  that  possess  adequate  knowledge  of  their  career  path. 


2  . 

Educate  the  Workforce 

a) 

Activity:  Establish 

training 

and 

education 

opportunities . 

Metrics:  Measure  the  number  of  acquisition  related 
training  and  education  opportunities  available  to  the 
acquisition  workforce.  Compare  this  number  to  pre-DAWIA 
levels  if  the  data  is  available.  Another  metric  presents  the 
amount  (measured  in  hours,  classes,  degrees,  etc.)  of 
education  and  training  acquisition  workforce  members  have 
received  before  DAWIA  and  after  DAWIA.  A  metric  proposed  by 
the  ARBG  in  the  previous  chapter  (Fig.  5)  measures  workforce 
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quality  by  presenting  the  percentage  of  acquisition  certified 
personnel . 

b)  Activity:  Conduct  an  intern  program  to  provide 
individuals  opportunities  for  accelerated  promotions,  career 
broadening  assignments,  and  specified  training  to  prepare  them 
for  entry  into  the  Acquisition  Corps. [Ref.  7:  p.  1651] 

Metrics :  Measure  the  percentage  of  workforce  members  that 
enter  and  complete  intern  programs.  Compare  these  figures  to 
pre-DAWIA  figures  (if  available) .  Measure  career  progression 
of  individuals  following  completion  of  intern  programs  by 
measuring  the  average  pay  increase,  grade  increase,  or  job 
advancement  and  compare  to  pre-DAWIA  figures  (if  available) . 

c)  Activity:  Conduct  a  cooperative  education  credit 
program  in  which  accredited  institutions  of  higher  education 
grant  undergraduate  credit  to  workforce  members  for  work 
performed  in  acquisition  positions . [Ref .  7:  p.  1652] 

Metrics:  Measure  the  number  of  educational  institutions 
participating  in  the  cooperative  program  and  compare  to  pre- 
DAWIA.  Measure  the  percentage  of  workforce  members 
participating  in  cooperative  programs  and  compare  to  pre- 
DAWIA. 

d)  Activity:  Establish  a  scholarship  program  for  the 
purpose  of  qualifying  personnel  for  acquisition 
positions . [Ref .  7:  p.  1651] 

Metrics:  Use  a  go/no-go  metric  to  determine  whether  a 
scholarship  program  has  been  established.  Measure  the 
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percentage  of  acquisition  personnel  participating  in  the 
program.  Measure  the  career  progression  of  personnel  that 
complete  the  program. 

e)  Activity:  Provide  tuition  reimbursement  and  student 
loan  repayment  for  acquisition  personnel. [Ref.  7:  p.  1653] 

Metrics:  Measure  the  percentage  of  acquisition  personnel 
that  receive  these  forms  of  payment.  Compare  to  the  pre-DAWIA 
percentage  (if  available) . 

3.  Increase  the  Number  of  Senior  Level  Civilian 
Acquisition  Positions 

a)  Activity:  Increase  the  number  of  senior  level  civilian 
acquisition  positions. 

Metrics:  Measure  the  percentage  of  senior  level  positions 
civilians  are  eligible  to  occupy.  Compare  this  percentage  to 
the  pre-DAWIA  percentage.  Also  measure  the  actual  civilian 
occupancy  of  those  positions  and  compare  to  pre-DAWIA. 

b)  Activity:  Provide  civilian  leadership  and  management 
training . 

Metrics:  Measure  the  quantity  of  civilian  leadership  and 
management  training  offered.  Measure  the  percentage  of  the 
civilian  acquisition  workforce  that  have  attended  the 
training.  Compare  these  figures  to  pre-DAWIA  figures  (if 
available) . 

4 .  Provide  Career  Growth 

a)  Activity:  Provide  a  career  growth  model. 

Metric:  Measure  career  growth  of  acquisition  workforce 
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members  by  tracking  promotion  rates,  pay  increases,  job 
advancement,  etc.  Compare  these  numbers  to  pre-DAWIA  figures 
(if  available) . 

5.  Grow  Program  Managers  (PMs) 

a)  Activity:  Establish  career  paths  and 
training/education  opportunities  that  lead  to  PM  selection  for 
acquisition  workforce  members. 

Metrics :  Measure  the  percentage  of  PMs  that  began  at  the 
most  junior  level  within  the  acquisition  workforce  and  compare 
to  pre-DAWIA  percentage.  Measure  the  average  amount  of  PM 
acquisition  experience,  education  and  training  and  compare  to 
the  pre-DAWIA  average.  Measure  the  promotion  rates  of 
military  acquisition  members  and  compare  to  the  promotion 
rates  of  non-acquisition  military.  Compare  results  to  those 
results  prior  to  DAWIA. 

b)  Activity:  Provide  opportunities  for  job  variability. 

Metrics:  To  grow  PMs,  ideally  a  system  would  be 

established  to  expose  workforce  members  to  various  areas  of 
acquisition (i .e.  program  management,  testing,  requirements 
development,  logistics,  etc.).  Therefore,  we  can  measure  this 
activity  by  tracking  variability  in  job  areas  and  perform 
ongoing  trend  analyses . 

6 .  Stabilize  Programs  Through  Reduction  in  PM  Turnover 

a)  Activity:  Require  PMs  to  remain  in  position  until 
completion  of  the  major  milestone  that  occurs  closest  to  the 
PM's  four  year  mark. 
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Metrics:  Measure  average  time  an  occupant  serves  as  a  PM. 
Compare  to  the  pre-DAWIA  time.  Measure  the  average  number  of 
PMs  for  all  programs  from  program  conception  to  fielding  of 
the  system  and  compare  to  pre-DAWIA  average. 

7 .  Increase  Acquisition  Expertise  and  Experience 

a)  Activity:  All  of  the  previous  activities  support  this 
particular  objective.  We  must  determine  if  the  above 
activities  result  in  increased  expertise  and  experience. 

Metrics :  Measure  expertise  through  the  use  of  a 
standardized  test.  Link  expertise  to  workforce  membership  in 
professional  associations  that  have  qualification  requirements 
such  as  exam  requirements,  experience,  etc.  Measure  the 
number  of  members.  However,  if  professional  association 
membership  is  voluntary,  this  metric  can  be  misleading  and 
offer  little  indication  of  workforce  expertise.  An 
alternative  could  be  establishing  a  DoD  professional 
association  that  requires  personnel  occupying  certain  critical 
acquisition  positions  (i.e.  Program  Managers)  to  certify 
through  use  of  an  exam  or  other  criteria. 

If  we  assume  a  positive  relationship  exists  between 
expertise  and  promotion,  measure  the  promotion  rates  of  the 
workforce  and  compare  to  pre-DAWIA.  To  measure  experience, 
measure  the  average  number  of  years  of  acquisition  experience 
of  each  workforce  member  and  compare  to  pre-DAWIA. 

8.  Control  Program  Costs 

a)  Activity:  Create  a  quality  workforce  through 
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training,  education,  and  experience  that  results  in 
improvement  in  program  cost  control . 

Metric:  The  objective  of  any  relevant  metric  would  be  to 
link  quality  of  personnel  to  program  cost  control.  As  a 
result,  measure  the  training,  education,  and  experience  levels 
of  personnel  who  served  on  programs  that  controlled  costs . 
Compare  these  figures  to  programs  that  did  not  control  costs . 
In  the  comparison  look  for  any  trends  that  relate  quality  of 
personnel  (i.e.  meets  certification  requirements)  to  program 
cost  control . 

9.  Efficiently  Acquire  Parts,  Supplies,  Materials,  and 
Programs 

This  particular  objective  is  the  ultimate  objective  of 
acquisition  reform  and  DAWIA.  All  of  the  above  DAWIA 
objectives  and  activities  are  a  subset  of  this  objective. 
What's  needed  in  order  to  develop  activities  and  metrics  for 
this  objective  is  to  establish  a  link  between  DAWIA 
(objectives  and  activities)  and  improved  efficiencies  in 
acquiring  parts,  supplies,  materials,  and  programs. 

a)  Activity:  Establish  a  quality  workforce  by  mandating 
training,  education,  and  experience  requirements  that  result 
in  improved  efficiencies  when  acquiring  parts,  supplies, 
materials,  and  programs. 

Metrics:  To  measure,  we  begin  with  the  acquisition  reform 
metrics  identified  in  the  previous  chapter  (Figures  1-31)  to 
determine  whether  the  acquisition  process  has  shown  efficiency 
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improvement . 

The  next  step  to  establishing  metrics  for  this  activity 
is  to  determine  what  impact  a  higher  quality  workforce  would 
have  on  an  improved  efficiency.  This  can  be  a  challenge  since 
an  improved  efficiency  could  result  from  many  factors  other 
than  or  in  addition  to  training  and  education,  as  mentioned  in 
the  previous  chapter. 

However,  many  of  these  factors  may  indirectly  involve 
training  and  education.  A  number  of  the  enterprise  metrics, 
as  defined  by  the  ARBG,  attempt  to  measure  improvements 
resulting  from  various  initiatives  i.e.  performance 
specifications,  IPTs,  commercial  specifications,  etc. 

In  order  for  the  initiatives  to  be  effective,  they  must 
be  properly  implemented.  Proper  implementation  often  requires 
a  sufficient  amount  of  training  and  education.  Therefore,  a 
useful  metric  could  measure  the  quantity  or  percentage  of  the 
acquisition  workforce  that  has  been  trained  on  a  particular 
initiative.  Additionally,  this  type  of  metric  could  be  an 
indicator  of  an  initiative's  implementation  and  effectiveness. 
This  type  of  metric  links  training  and  education  to  other 
reform  initiatives. 

For  example,  proponents  for  the  use  of  IPTs  expect  to  see 
improved  efficiencies  (i.e.  reduced  acquisition  cycle  time) 
resulting  from  their  use.  Obviously,  there  are  many  factors 
that  could  determine  the  success  of  an  IPT.  However,  one 
cannot  ignore  the  importance  of  training  workforce  members  in 
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the  proper  use  of  IPTs.  Thus,  we  would  measure  the  percentage 
of  the  workforce  that  has  received  adequate  (adequate  would 
have  to  be  defined)  IPT  training.  If  the  training  numbers  are 
low  and  we  see  little  improvement  in  efficiencies  that  result 
from  IPT  use,  then  there  could  be  a  correlation. 

If  the  training  numbers  are  high  and  there  is  little 
efficiency  improvement,  reasons  could  be  inadequate  training, 
inadequate  implementation,  or  ineffective  concept.  Regardless 
of  reason,  use  of  this  metric  could  help  determine  why  an 
initiative  is  failing  to  reach  expectations  and  thus  help 
leaders  direct  future  efforts. 

There  could  be  other  effects  of  a  better  quality 
workforce  on  the  acquisition  process  that  are  worth 
considering.  These  effects  are  usually  stated  in  terms  of 
organizational  benefits  and,  in  some  cases,  can  be  translated 
into  improvements  in  cost,  schedule,  and  performance. 

Some  of  the  benefits  identified  below  can  be  used 
directly  as  metrics  since  they  are  quantifiable.  Other 
benefits  are  not  quantifiable  and  in  order  to  be  used  as 
metrics  they  must  be  converted  into  measurable  form.  This 
measurable  form  can  be  a  rating,  of  say,  1,  most  negative,  to 
10,  most  positive  (the  difficulty  with  using  rating  metrics  is 
that  you  can't  go  back  in  time  to  collect  data  on  these  type 
of  metrics  if  similar  ratings  were  not  used) .  Thus  the  list 
of  benefits  that  follow  (Table  l)can  be  used  as  direct  metrics 
and  rating  metrics  [Ref.  12:  p.  5] : 
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Decreases  or  reductions  in 

Increases  or  improvements  in 

Absentee  rate 

Attitude  rating  towards  the 
enterprise 

Break-in  time  for  new  hires 

Communication  skill  rating 

Customer  complaints 

Leadership  rating 

Employee  errors 

Customer  relation  rating 

Employee  tension  rating 

Customer  satisfaction  rating 

Grievances 

Employee  attitude  rating 

Labor  disputes 

Employee  job  satisfaction 
rating 

Lost  time 

Employee  judgement  rating 

Misfits 

Employee  morale  rating 

Misinterpretations  of  policy 

Employee  motivation  rating 

Operating  costs 

Employee  suggestions 

Rejects  and  reworks 

Employee  work  habit  rating 

Requirements  for  overtime 

Enterprise  flexibility  and 
adaptability  to  changing 
conditions  rating 

Sick-leave  rates 

Human  relations  skills 
rating 

Tardiness  rates 

Labor  relations  rating 

Time  required  to  introduce 
new  products  or  processes 

Quality  of  management  and 
supervision  rating 

Turnover  rate 

Rating  for  the  understanding 
of  organization  goals, 
plans,  policies,  procedures 

Unit  costs 

Violations  of  rules  and 
regulations 

Work  backlog  rate 

Table  1.  After  Ref.  [12] 
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D .  CHAPTER  SUMMARY 

The  objectives  of  DAWIA  focus  on  two  major  areas --improve 
the  quality  of  the  workforce  and  improve  the  acquisition 
process.  This  study  used  a  methodology  for  developing  metrics 
for  these  objectives.  This  chapter  introduced  metrics  for 
measuring  the  quality  of  the  workforce  and  metrics  for  linking 
the  quality  of  the  workforce  to  an  improved  acquisition 
process.  These  metrics  can  help  to  determine  how  well  DAWIA 
has  been  implemented  and  how  effective  DAWIA  has  been  towards 
improving  the  quality  of  the  workforce  and  the  acquisition 
process.  However,  there  is  still  a  continuing  need  for 
research  to  analyze  and  refine  metrics  for  DAWIA. 
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V. 


CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

Various  reform  initiatives  have  attempted  to  improve  the 
acquisition  process .  The  success  or  failure  of  these 
initiatives  have  often  been  based  on  subjective 
determinations.  In  order  to  determine  the  true  effect  of 
these  initiatives,  we  must  be  able  to  use  metrics  to  measure 
the  implementation  and  effect  of  the  initiative  on  the 
acquisition  process . 

DoD,  through  the  establishment  of  the  Acquisition  Reform 
Baseline  Group  (ARBG) ,  has  initiated  efforts  to  develop 
metrics  for  measuring  the  effect  of  acquisition  reform 
initiatives.  However,  only  one  of  the  ARBG  proposed  metrics 
relates  to  DAWIA.  This  metric,  the  acquisition  workforce 
metric,  measures  the  certification  percentages  of  the 
acquisition  workforce.  This  metric  is  primarily  linked  to 
DAWIA' s  objective  of  educating  the  workforce. 

This  study  identified  the  specific  objectives  of  DAWIA. 
This  study  used  the  Policy  Effectiveness  Model  to  analyze 
DAWIA' s  objectives  and  introduce  metrics  that  can  possibly  be 
used  to  measure  these  objectives.  Use  of  these  metrics  could 
ultimately  result  in  an  ability  to  measure  the  effectiveness 
of  DAWIA. 
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B .  RECOMMENDATIONS 

Based  on  the  history  of  acquisition  reform,  the  benefits 
of  metrics,  and  the  tools  available  for  developing  metrics, 
this  study  recommends  that  metrics  continue  to  be  developed 
and  refined  to  measure  the  implementation  and  effectiveness  of 
acquisition  reform  initiatives.  Metrics  development  should 
occur  prior  to  implementation  of  initiative  using  the 
following  steps: 

1.  Determine  the  initiative's  objectives. 

2.  Determine  the  activities  for  accomplishing  the 
objectives . 

3.  Develop  metrics  for  measuring  the  activities. 

4.  Implement  the  initiative. 

5.  Measure  the  activities  using  the  metrics. 

6 .  Analyze  the  measurements  and  make  an  assessment  on 
the  initiative's  implementation  and  effectiveness. 

7.  Make  changes  based  on  the  assessment. 

This  study  also  recommends  that  the  proposed  ARBG  metrics 
be  expanded  to  include  DAWIA  metrics.  Also  redefine  some  of 
the  ARBG  metrics  so  that  they  consider  the  impact  of  a  higher 
quality  workforce.  Improved  efficiencies  result  not  only  from 
initiatives  but  also  from  an  improved  workforce. 

Finally,  this  study  recommends  that  DAWIA  metrics  be 
linked  to  acquisition  reform  by  establishing  metrics  to 
monitor  workforce  training  on  reform  initiatives.  Effective 
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implementation  of  initiatives  requires  adequate  training.  In 
order  to  properly  assess  implementation  of  a  reform  initiative 
it  is  necessary  to  measure  workforce  training  on  the 
initiative . 

C.  ANSWERS  TO  RESEARCH  QUESTIONS 

1.  How  can  DAWIA's  effectiveness  be  measured? 

Begin  by  determining  the  objectives  of  DAWIA.  Next, 
determine  the  activities  that  will  have  to  be  accomplished  in 
order  to  meet  those  objectives.  Finally,  develop  metrics  for 
measuring  those  activities. 

2.  What  are  the  specific  objectives  of  DAWIA? 

The  specific  objectives  of  DAWIA  are  as  follows: 

-  Provide  a  clear  career  track  for  the  acquisition 
workforce 

-  Educate  the  workforce 

-  Increase  the  number  of  civilian  acquisition  positions 

-  Provide  career  growth 

-  Grow  PMs 

-  Stabilize  programs  through  reduction  in  PM  turnover 
The  aim  of  the  above  objectives  is  to  achieve  the 

following  ultimate  objectives: 

-  Increase  acquisition  expertise  and  experience 

-  Control  program  costs 

-  Efficiently  acquire  parts,  supplies,  materials,  and 
programs 

3 .  What  is  a  metric  and  what  types  of  metrics  are 
suitable  for  acquisition  reform? 

Metrics  are  quantifiable  measures  that  allow  us  measure 
people,  processes,  and  products.  Suitable  metrics  for 
acquisition  reform  enable  us  to  measure  the  implementation  and 
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effectiveness  of  acquisition  reform  initiatives  thus  allowing 
us  to  assess  the  initiative's  impact  on  the  acquisition 
process . 

4 .  What  metrics  are  currently  being  used  to  measure  the 
effects  of  DAWIA  on  the  acquisition  process? 

Metrics  are  not  being  used  or  proposed  to  directly 
measure  DAWIA' s  effect  on  the  acquisition  process.  The  only 
DAWIA  metrics  currently  proposed  measures  the  effectiveness  of 
the  qualification  requirements  on  the  workforce  by  measuring 
workforce  certification  percentages. 

5.  Are  these  appropriate  metrics? 

This  metric  is  appropriate,  but  is  very  limited  in  its 
usefulness.  If  the  only  objective  of  DAWIA  was  simply  to 
increase  the  education  levels  of  the  acquisition  workforce, 
then  this  metric  would  completely  serve  the  purpose.  However, 
DAWIA  has  other  objectives  that  not  only  aim  to  improve  the 
quality  of  the  workforce  but  the  quality  of  the  acquisition 
process.  Thus,  there  is  a  need  to  expand  DAWIA  metrics  in 
order  to  measure  and  assess  DAWIA' s  impact  on  the  acquisition 
process . 

6.  Are  there  standardized  metrics  for  acquisition 
reform  initiatives?  If  not,  should  standards  be  established? 

The  DoD  Acquisition  Reform  Baseline  Group  has  proposed  a 
set  of  metrics  that  appear  to  be  headed  towards  a  standard  for 
measuring  the  effects  of  acquisition  reform.  These  metrics 
should  be  standardized  with  consideration  for  adding  the 
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DAWIA  metrics  proposed  in  Chapter  IV  of  this  study. 

D.  RECOMMENDATIONS  FOR  FURTHER  STUDY 

This  study  is  the  initial  step  towards  developing 
measures  of  effectiveness  for  DAWIA  and  other  acquisition 
reform  initiatives.  There  is  more  work  to  be  done.  Following 
are  recommendations  for  study  that  will  further  advance 
efforts  towards  measuring  the  effectiveness  of  acquisition 
reform: 

1.  Further  refine  DAWIA  metric  design. 

2 .  Conduct  tests  of  significance  for  DAWIA  metrics  on 
the  acquisition  process. 

3 .  Measure  and  assess  the  effectiveness  of  DAWIA  by 
applying  the  metrics  developed  in  this  study. 

4 .  Perform  similar  studies  in  other  areas  of 
acquisition  reform  by  applying  the  methodology  used  in  this 
study.  This  should  lead  to  the  development  of  metrics  for 
measuring  the  implementation  and  effectiveness  of  other 
Government  reform  initiatives. 
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